Method and system for periodic trace sampling for real-time generation of segments of call stack trees

ABSTRACT

A method and system for profiling a program using periodic trace sampling is provided. During the execution of the program, sample-based profiling of the executing program is performed—for a predetermined period, a profiler performs trace processing for the program, after which the profiler pauses and does not perform trace processing for a predetermined period. The periods controlling the profiler may be selected by a user, and the periods may be measured by temporal or non-temporal metrics. The profiler cycles through these periods, during which selected events are processed to generate a profile of the execution flows within the program. For each sample period, a tree data structure is generated in which nodes of the tree data structure represent the routines of the program that execute during the sample period, as may be indicated by entry and exit events caused by the execution of the routines. When the execution of the program is complete, the tree data structures from each sample period are merged into a resulting tree data structure.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of the following and commonlyassigned applications entitled “SYSTEM AND METHOD FOR PROVIDING TRACEINFORMATION REDUCTION”, U.S. application Ser. No. 08/989,725, filed onDec. 12, 1997, now U.S. Pat. No. 6,055,492; “A METHOD AND APPARATUS FORSTRUCTURED PROFILING OF DATA PROCESSING SYSTEMS AND APPLICATIONS”, U.S.application Ser. No. 09/052,329, filed on Mar. 31, 1998, now U.S. Pat.No. 6,002,872; “METHOD AND APPARATUS FOR PROFILING PROCESSES IN A DATAPROCESSING SYSTEM”, U.S. application Ser. No. 09/177,031, filed on Oct.22, 1998, now U.S. Pat. No. 6,311,325; “PROCESS AND SYSTEM FOR MERGINGTRACE DATA FOR PRIMARILY INTERPRETED METHODS”, U.S. application Ser. No.09/343,439, filed on Jun. 30, 1999; now U.S. Pat. No. 6,553,564 and“METHOD AND SYSTEM FOR MERGING EVENT-BASED DATA AND SAMPLED DATA INTOPOSTPROCESSED TRACE OUTPUT”, U.S. application Ser. No. 09/343,438, filedon Jun. 30, 1999 now U.S. Pat. No. 6,513,155.

Additionally, this application is related to U.S. patent applicationSer. No. 09/052,331, filed Mar. 31, 1998, which issued as U.S. Pat. No.6,158,024 and is hereby incorporated by reference.

The present invention is related to the following application entitled“METHOD AND SYSTEM FOR DETECTING AND RECOVERING FROM ERRORS IN TRACEDATA”, U.S. application Ser. No. 09/393,083, filed Sep. 9, 1999; and“METHOD AND SYSTEM FOR PERIODIC TRACE SAMPLING FOR REAL-TIME GENERATIONOF SEGMENTS OF CALL STACK TREES AUGMENTED WITH CALL STACK POSITIONDETERMINATION”, U.S. application Ser. No. 09/418,378, filed Oct. 14,1999; currently pending, and assigned to the same assignee.

BACKGROUND OF THE INVENTION

1. Technical Field

The present invention relates to an improved data processing system and,in particular, to a method and apparatus for optimizing performance in adata processing system. Still more particularly, the present inventionprovides a method and apparatus for a software program development toolfor enhancing performance of a software program through softwareprofiling.

2. Description of Related Art

In analyzing and enhancing performance of a data processing system andthe applications executing within the data processing system, it ishelpful to know which software modules within a data processing systemare using system resources. Effective management and enhancement of dataprocessing systems requires knowing how and when various systemresources are being used. Performance tools are used to monitor andexamine a data processing system to determine resource consumption asvarious software applications are executing within the data processingsystem. For example, a performance tool may identify the most frequentlyexecuted modules and instructions in a data processing system, or mayidentify those modules which allocate the largest amount of memory orperform the most I/O requests. Hardware performance tools may be builtinto the system or added at a later point in time. Software performancetools also are useful in data processing systems, such as personalcomputer systems, which typically do not contain many, if any, built-inhardware performance tools.

One known software performance tool is a trace tool. A trace tool mayuse more than one technique to provide trace information that indicatesexecution flows for an executing program. One technique keeps track ofparticular sequences of instructions by logging certain events as theyoccur, so-called event-based profiling technique. For example, a tracetool may log every entry into, and every exit from, a module,subroutine, method, function, or system component. Alternately, a tracetool may log the requester and the amounts of memory allocated for eachmemory allocation request. Typically, a time-stamped record is producedfor each such event. Corresponding pairs of records similar toentry-exit records also are used to trace execution of arbitrary codesegments, starting and completing I/O or data transmission, and for manyother events of interest.

In order to improve performance of code generated by various families ofcomputers, it is often necessary to determine where time is being spentby the processor in executing code, such efforts being commonly known inthe computer processing arts as locating “hot spots.” Ideally, one wouldlike to isolate such hot spots at the instruction and/or source line ofcode level in order to focus attention on areas which might benefit mostfrom improvements to the code.

Another trace technique involves periodically sampling a program'sexecution flows to identify certain locations in the program in whichthe program appears to spend large amounts of time. This technique isbased on the idea of periodically interrupting the application or dataprocessing system execution at regular intervals, so-called sample-basedprofiling. At each interruption, information is recorded for apredetermined length of time or for a predetermined number of events ofinterest. For example, the program counter of the currently executingthread, which is a process that is part of the larger program beingprofiled, may be recorded during the intervals. These values may beresolved against a load map and symbol table information for the dataprocessing system at post-processing time, and a profile of where thetime is being spent may be obtained from this analysis.

For example, isolating such hot spots to the instruction level permitscompiler writers to find significant areas of suboptimal code generationat which they may thus focus their efforts to improve code generationefficiency. Another potential use of instruction level detail is toprovide guidance to the designer of future systems. Such designersemploy profiling tools to find characteristic code sequences and/orsingle instructions that require optimization for the available softwarefor a given type of hardware.

Continuous event-based profiling has limitations. For example,continuous event-based profiling is expensive in terms of performance(an event per entry and per exit), which can and often does perturb theresulting view of performance. Additionally, this technique is notalways available because it requires the static or dynamic insertion ofentry/exit events into the code. This insertion of events is sometimesnot possible or is often difficult. For example, if source code isunavailable for the to-be-instrumented code, event-based profiling maynot be feasible. However, it is possible to instrument an interpreter ofthe source code to obtain event-base profiling information withoutchanging the source code.

On the other hand, sample-based profiling provides only a “flat view” ofsystem performance but does provide the benefits of reduced cost andreduced dependence on hooking-capability. Further, sample-basedtechniques do not identify where the time is spent in many small andseemingly unrelated functions or in situations in which no clear hotspot is apparent. Without an understanding of the program structure, itis not clear with a “flat” profile how to determine where theperformance improvements can be obtained.

Therefore, it would be advantageous to provide a system that combinesthe benefits of event-based profiling with the benefits of reducedsystem perturbation in sample-based profiling. It would be particularlyadvantageous to provide the ability to enable and disable profiling ofselected portions of a data processing system and to combine the outputfrom different profiling periods into a single merged presentation.

SUMMARY OF THE INVENTION

The present invention provides a method and system for profiling aprogram using periodic trace sampling. During the execution of theprogram, sample-based profiling of the executing program isperformed—for a predetermined period, a profiler performs traceprocessing for the program, after which the profiler pauses and does notperform trace processing for a predetermined period. The periodscontrolling the profiler may be selected by a user, and the periods maybe measured by temporal or non-temporal metrics. The profiler cyclesthrough these periods, during which selected events are processed togenerate a profile of the execution flows within the program. For eachsample period, a tree data structure is generated in which nodes of thetree data structure represent the routines of the program that executeduring the sample period, as may be indicated by entry and exit eventscaused by the execution of the routines. When the execution of theprogram is complete, the tree data structures from each sample periodare merged into a resulting tree data structure.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features believed characteristic of the invention are setforth in the appended claims. The invention itself, however, as well asa preferred mode of use, further objectives and advantages thereof, willbest be understood by reference to the following detailed description ofan illustrative embodiment when read in conjunction with theaccompanying drawings, wherein:

FIG. 1 depicts a distributed data processing system in which the presentinvention may be implemented;

FIGS. 2A-B are block diagrams depicting a data processing system inwhich the present invention may be implemented;

FIG. 3A is a block diagram depicting the relationship of softwarecomponents operating within a computer system that may implement thepresent invention;

FIG. 3B is a block diagram depicting a Java virtual machine inaccordance with a preferred embodiment of the present invention;

FIG. 4 is a block diagram depicting components used to profile processesin a data processing system;

FIG. 5 is an illustration depicting various phases in profiling theactive processes in an operating system;

FIG. 6 is a flowchart depicting a process used by a trace program forgenerating trace records from processes executing on a data processingsystem;

FIG. 7 is a flowchart depicting a process used in a system interrupthandler trace hook;

FIG. 8 is a diagram depicting the call stack containing stack frames;

FIG. 9 is an illustration depicting a call stack sample;

FIG. 10A is a diagram depicting a program execution sequence along withthe state of the call stack at each function entry/exit point;

FIG. 10B is a diagram depicting a particular timer based sampling of theexecution flow depicted in FIG. 10A;

FIGS. 10C-D are time charts providing an example of the types of timefor which the profiling tool accounts;

FIG. 11A is a diagram depicting a tree structure generated from samplinga call stack;

FIG. 11B is a diagram depicting an event tree which reflects call stacksobserved during system execution;

FIG. 12 is a table depicting a call stack tree;

FIG. 13 is a flow chart depicting a method for building a call stacktree using a trace text file as input;

FIG. 14 is a flow chart depicting a method for building a call stacktree dynamically as tracing is taking place during system execution;

FIG. 15 is a diagram depicting a record generated using the processes ofpresent invention;

FIG. 16 is a diagram depicting another type of report that may beproduced to show the calling structure between routines shown in FIG.12;

FIG. 17 is a table depicting a report generated from a trace filecontaining both event-based profiling information (method entry/exits)and sample-based profiling information (stack unwinds);

FIGS. 18A-18B are tables depicting major codes and minor codes that maybe employed to instrument modules for profiling;

FIG. 19 is a flowchart depicting the processing of exit events,including error recovery processing, in an execution flow that maycontain errors in the form of mismatched events;

FIG. 20 is a flowchart depicting a process for sample-based profiling ofentry/exit events employing user-specified sample metrics to buildindependent call stack tree segments; and

FIG. 21 is a flowchart depicting the process by which one tree, calledthe source tree, is added to a second tree, called the target tree, toobtain a union between the two trees.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

With reference now to the figures, and in particular with reference toFIG. 1, a pictorial representation of a distributed data processingsystem in which the present invention may be implemented is depicted.

Distributed data processing system 100 is a network of computers inwhich the present invention may be implemented. Distributed dataprocessing system 100 contains a network 102, which is the medium usedto provide communications links between various devices and computersconnected together within distributed data processing system 100.Network 102 may include permanent connections, such as wire or fiberoptic cables, or temporary connections made through telephoneconnections.

In the depicted example, a server 104 is connected to network 102 alongwith storage unit 106. In addition, clients 108, 110, and 112 also areconnected to a network 102. These clients 108, 110, and 112 may be, forexample, personal computers or network computers. For purposes of thisapplication, a network computer is any computer, coupled to a network,which receives a program or other application from another computercoupled to the network. In the depicted example, server 104 providesdata, such as boot files, operating system images, and applications toclients 108-112. Clients 108, 110, and 112 are clients to server 104.Distributed data processing system 100 may include additional servers,clients, and other devices not shown. In the depicted example,distributed data processing system 100 is the Internet with network 102representing a worldwide collection of networks and gateways that usethe TCP/IP suite of protocols to communicate with one another. At theheart of the Internet is a backbone of high-speed data communicationlines between major nodes or host computers, consisting of thousands ofcommercial, government, educational, and other computer systems, thatroute data and messages. Of course, distributed data processing system100 also may be implemented as a number of different types of networks,such as, for example, an Intranet or a local area network.

FIG. 1 is intended as an example, and not as an architectural limitationfor the processes of the present invention.

With reference now to FIG. 2A, a block diagram of a data processingsystem which may be implemented as a server, such as server 104 in FIG.1, is depicted in accordance to the present invention. Data processingsystem 200 may be a symmetric multiprocessor (SMP) system including aplurality of processors 202 and 204 connected to system bus 206.Alternatively, a single processor system may be employed. Also connectedto system bus 206 is memory controller/cache 208, which provides aninterface to local memory 209. I/O Bus Bridge 210 is connected to systembus 206 and provides an interface to I/O bus 212. Memorycontroller/cache 208 and I/O Bus Bridge 210 may be integrated asdepicted.

Peripheral component interconnect (PCI) bus bridge 214 connected to I/Obus 212 provides an interface to PCI local bus 216. A modem 218 may beconnected to PCI local bus 216. Typical PCI bus implementations willsupport four PCI expansion slots or add-in connectors. Communicationslinks to network computers 108-112 in FIG. 1 may be provided throughmodem 218 and network adapter 220 connected to PCI local bus 216 throughadd-in boards.

Additional PCI bus bridges 222 and 224 provide interfaces for additionalPCI buses 226 and 228, from which additional modems or network adaptersmay be supported. In this manner, server 200 allows connections tomultiple network computers. A memory mapped graphics adapter 230 andhard disk 232 may also be connected to I/O bus 212 as depicted, eitherdirectly or indirectly.

Those of ordinary skill in the art will appreciate that the hardwaredepicted in FIG. 2A may vary. For example, other peripheral devices,such as optical disk drive and the like also may be used in addition orin place of the hardware depicted. The depicted example is not meant toimply architectural limitations with respect to the present invention.

The data processing system depicted in FIG. 2A may be, for example, anIBM RISC/System 6000 system, a product of International BusinessMachines Corporation in Armonk, New York, running the AdvancedInteractive Executive (AIX) operating system.

With reference now to FIG. 2B, a block diagram of a data processingsystem in which the present invention may be implemented is illustrated.Data processing system 250 is an example of a client computer. Dataprocessing system 250 employs a peripheral component interconnect (PCI)local bus architecture. Although the depicted example employs a PCI bus,other bus architectures such as Micro Channel and ISA may be used.Processor 252 and main memory 254 are connected to PCI local bus 256through PCI Bridge 258. PCI Bridge 258 also may include an integratedmemory controller and cache memory for processor 252. Additionalconnections to PCI local bus 256 may be made through direct componentinterconnection or through add-in boards. In the depicted example, localarea network (LAN) adapter 260, SCSI host bus adapter 262, and expansionbus interface 264 are connected to PCI local bus 256 by direct componentconnection. In contrast, audio adapter 266, graphics adapter 268, andaudio/video adapter (A/V) 269 are connected to PCI local bus 266 byadd-in boards inserted into expansion slots. Expansion bus interface 264provides a connection for a keyboard and mouse adapter 270, modem 272,and additional memory 274. SCSI host bus adapter 262 provides aconnection for hard disk drive 276, tape drive 278, and CD-ROM 280 inthe depicted example. Typical PCI local bus implementations will supportthree or four PCI expansion slots or add-in connectors.

An operating system runs on processor 252 and is used to coordinate andprovide control of various components within data processing system 250in FIG. 2B. The operating system may be a commercially availableoperating system such as JavaOS For Business™ or OS/2™which areavailable from International Business Machines Corporation™. JavaOS isloaded from a server on a network to a network client and supports Javaprograms and applets. A couple of characteristics of JavaOS that arefavorable for performing traces with stack unwinds, as described below,are that JavaOS does not support paging or virtual memory. An objectoriented programming system such as Java may run in conjunction with theoperating system and may provide calls to the operating system from Javaprograms or applications executing on data processing system 250.Instructions for the operating system, the object-oriented operatingsystem, and applications or programs are located on storage devices,such as hard disk drive 276 and may be loaded into main memory 254 forexecution by processor 252. Hard disk drives are often absent and memoryis constrained when data processing system 250 is used as a networkclient.

Those of ordinary skill in the art will appreciate that the hardware inFIG. 2B may vary depending on the implementation. For example, otherperipheral devices, such as optical disk drives and the like may be usedin addition to or in place of the hardware depicted in FIG. 2B. Thedepicted example is not meant to imply architectural limitations withrespect to the present invention. For example, the processes of thepresent invention may be applied to a multiprocessor data processingsystem.

The present invention provides a process and system for profilingsoftware applications. Although the present invention may operate on avariety of computer platforms and operating systems, it may also operatewithin a Java runtime environment. Hence, the present invention mayoperate in conjunction with a Java virtual machine (JVM) yet within theboundaries of a JVM as defined by Java standard specifications. In orderto provide a context for the present invention, portions of theoperation of a JVM according to Java specifications are hereindescribed.

With reference now to FIG. 3A, a block diagram illustrates therelationship of software components operating within a computer systemthat may implement the present invention. Java-based system 300 containsplatform specific operating system 302 that provides hardware and systemsupport to software executing on a specific hardware platform. JVM 304is one software application that may execute in conjunction with theoperating system. JVM 304 provides a Java run-time environment with theability to execute Java application or applet 306, which is a program,servlet, or software component written in the Java programming language.The computer system in which JVM 304 operates may be similar to dataprocessing system 200 or computer 100 described above. However, JVM 304may be implemented in dedicated hardware on a so-called Java chip,Java-on-silicon, or Java processor with an embedded picoJava core.

At the center of a Java run-time environment is the JVM, which supportsall aspects of Java's environment, including its architecture, securityfeatures, mobility across networks, and platform independence.

The JVM is a virtual computer, i.e. a computer that is specifiedabstractly. The specification defines certain features that every JVMmust implement, with some range of design choices that may depend uponthe platform on which the JVM is designed to execute. For example, allJVMs must execute Java bytecodes and may use a range of techniques toexecute the instructions represented by the bytecodes. A JVM may beimplemented completely in software or somewhat in hardware. Thisflexibility allows different JVMs to be designed for mainframe computersand PDAs.

The JVM is the name of a virtual computer component that actuallyexecutes Java programs. Java programs are not run directly by thecentral processor but instead by the JVM, which is itself a piece ofsoftware running on the processor. The JVM allows Java programs to beexecuted on a different platform as opposed to only the one platform forwhich the code was compiled. Java programs are compiled for the JVM. Inthis manner, Java is able to support applications for many types of dataprocessing systems, which may contain a variety of central processingunits and operating systems architectures. To enable a Java applicationto execute on different types of data processing systems, a compilertypically generates an architecture-neutral file format—the compiledcode is executable on many processors, given the presence of the Javarun-time system. The Java compiler generates bytecode instructions thatare nonspecific to a particular computer architecture. A bytecode is amachine independent code generated by the Java compiler and executed bya Java interpreter. A Java interpreter is part of the JVM thatalternately decodes and interprets a bytecode or bytecodes. Thesebytecode instructions are designed to be easy to interpret on anycomputer and easily translated on the fly into native machine code. Bytecodes are may be translated into native code by a just-in-time compileror JIT.

A JVM must load class files and execute the bytecodes within them. TheJVM contains a class loader, which loads class files from an applicationand the class files from the Java application programming interfaces(APIs) which are needed by the application. The execution engine thatexecutes the bytecodes may vary across platforms and implementations.

One type of software-based execution engine is a just-in-time compiler.With this type of execution, the bytecodes of a method are compiled tonative machine code upon successful fulfillment of some type of criteriafor jitting a method. The native machine code for the method is thencached and reused upon the next invocation of the method. The executionengine may also be implemented in hardware and embedded on a chip sothat the Java bytecodes are executed natively. JVMs usually interpretbytecodes, but JVMs may also use other techniques, such as just-in-timecompiling, to execute bytecodes.

Interpreting code provides an additional benefit. Rather thaninstrumenting the Java source code, the interpreter may be instrumented.Trace data may be generated via selected events and timers through theinstrumented interpreter without modifying the source code. Profileinstrumentation is discussed in more detail further below.

When an application is executed on a JVM that is implemented in softwareon a platform-specific operating system, a Java application may interactwith the host operating system by invoking native methods. A Java methodis written in the Java language, compiled to bytecodes, and stored inclass files. A native method is written in some other language andcompiled to the native machine code of a particular processor. Nativemethods are stored in a dynamically linked library whose exact form isplatform specific.

With reference now to FIG. 3B, a block diagram of a JVM is depicted inaccordance with a preferred embodiment of the present invention. JVM 350includes a class loader subsystem 352, which is a mechanism for loadingtypes, such as classes and interfaces, given fully qualified names. JVM350 also contains runtime data areas 354, execution engine 356, nativemethod interface 358, and memory management 374. Execution engine 356 isa mechanism for executing instructions contained in the methods ofclasses loaded by class loader subsystem 352. Execution engine 356 maybe, for example, Java interpreter 362 or just-in-time compiler 360.Native method interface 358 allows access to resources in the underlyingoperating system. Native method interface 358 may be, for example, aJava native interface.

Runtime data areas 354 contain native method stacks 364, Java stacks366, PC registers 368, method area 370, and heap 372. These differentdata areas represent the organization of memory needed by JVM 350 toexecute a program.

Java stacks 366 are used to store the state of Java method invocations.When a new thread is launched, the JVM creates a new Java stack for thethread. The JVM performs only two operations directly on Java stacks: itpushes and pops frames. A thread's Java stack stores the state of Javamethod invocations for the thread. The state of a Java method invocationincludes its local variables, the parameters with which it was invoked,its return value, if any, and intermediate calculations. Java stacks arecomposed of stack frames. A stack frame contains the state of a singleJava method invocation. When a thread invokes a method, the JVM pushes anew frame onto the Java stack of the thread. When the method completes,the JVM pops the frame for that method and discards it. The JVM does nothave any registers for holding intermediate values; any Java instructionthat requires or produces an intermediate value uses the stack forholding the intermediate values. In this manner, the Java instructionset is well-defined for a variety of platform architectures.

PC registers 368 are used to indicate the next instruction to beexecuted. Each instantiated thread gets its own pc register (programcounter) and Java stack. If the thread is executing a JVM method, thevalue of the pc register indicates the next instruction to execute. Ifthe thread is executing a native method, then the contents of the pcregister are undefined.

Native method stacks 364 store the state of invocations of nativemethods. The state of native method invocations is stored in animplementation-dependent way in native method stacks, registers, orother implementation-dependent memory areas. In some JVMimplementations, native method stacks 364 and Java stacks 366 arecombined.

Method area 370 contains class data while heap 372 contains allinstantiated objects. The JVM specification strictly defines data typesand operations. Most JVMs choose to have one method area and one heap,each of which are shared by all threads running inside the JVM. When theJVM loads a class file, it parses information about a type from thebinary data contained in the class file. It places this type informationinto the method area. Each time a class instance or array is created,the memory for the new object is allocated from heap 372. JVM 350includes an instruction that allocates memory space within the memoryfor heap 372 but includes no instruction for freeing that space withinthe memory. Memory management 374 in the depicted example manages memoryspace within the memory allocated to heap 370. Memory management 374 mayinclude a garbage collector which automatically reclaims memory used byobjects that are no longer referenced. Additionally, a garbage collectoralso may move objects to reduce heap fragmentation.

The processes within the following figures provide an overallperspective of the many processes employed within the present invention:processes that generate event-based profiling information; processesthat generate sample-based profiling information; processes that use theprofile information to generate more useful information, such asrepresentations of call stack trees, to be placed into profile reports;and processes that generate the profile reports for the user of theprofiling utility.

With reference now to FIG. 4, a block diagram depicts components used toprofile processes in a data processing system. A trace program 400 isused to profile processes 402. Trace program 400 may be used to recorddata upon the execution of a hook, which is a specialized piece of codeat a specific location in a routine or program in which other routinesmay be connected. Trace hooks are typically inserted for the purpose ofdebugging, performance analysis, or enhancing functionality. These tracehooks are employed to send trace data to trace program 400, which storesthe trace data in buffer 404. The trace data in buffer 404 may besubsequently stored in a file for post-processing, or the trace data maybe processed in real-time.

With Java operating systems, the present invention employs trace hooksthat aid in identifying methods that may be used in processes 402. Inaddition, since classes may be loaded and unloaded, these changes mayalso be identified using trace data. This is especially relevant with“network client” data processing systems, such as those that may operateunder JavaOS, since classes and jitted methods may be loaded andunloaded more frequently due to the constrained memory and role as anetwork client. Note that class load and unload information is alsorelevant in embedded application environments, which tend to be memoryconstrained.

With reference now to FIG. 5, a diagram depicts various phases inprofiling the processes active in an operating system. Subject to memoryconstraints, the generated trace output may be as long and as detailedas the analyst requires for the purpose of profiling a particularprogram.

An initialization phase 500 is used to capture the state of the clientmachine at the time tracing is initiated. This trace initialization dataincludes trace records that identify all existing threads, all loadedclasses, and all methods for the loaded classes. Records from trace datacaptured from hooks are written to indicate thread switches, interrupts,and loading and unloading of classes and jitted methods. Any class whichis loaded has trace records that indicate the name of the class and itsmethods. In the depicted example, four byte IDs are used as identifiersfor threads, classes, and methods. These IDs are associated with namesthat have been output in the records. A record is written to indicatewhen all of the start up information has been written.

Next, during the profiling phase 502, trace records are written to atrace buffer or trace file. In the present invention, a trace buffer mayhave a combination of types of records, such as those that may originatefrom a trace hook executed in response to a particular type of event,e.g., a method entry or method exit, and those that may originate from astack walking function executed in response to a timer interrupt, e.g.,a stack unwind record, also called a call stack record.

For example, the following process may occur during the profiling phaseif the user of the profiling utility has requested sample-basedprofiling information. Each time a particular type of timer interruptoccurs, a trace record is written, which indicates the system programcounter. This system program counter may be used to identify the routinethat is interrupted. In the depicted example, a timer interrupt is usedto initiate gathering of trace data. Of course, other types ofinterrupts may be used other than timer interrupts. Interrupts based ona programmed performance monitor event or other types of periodic eventsmay be employed.

In the post-processing phase 504, the data collected in the trace bufferis sent to a trace file for post-processing. In one configuration, thefile may be sent to a server, which determines the profile for theprocesses on the client machine. Of course, depending on availableresources, the post-processing also may be performed on the clientmachine. In post-processing phase 504, B-trees and/or hash tables may beemployed to maintain names associated the records in the trace file tobe processed. A hash table employs hashing to convert an identifier or akey, meaningful to a user, into a value for the location of thecorresponding data in the table. While processing trace records, theB-trees and/or hash tables are updated to reflect the current state ofthe client machine, including newly loaded jitted code or unloaded code.Also, in the post-processing phase 504, each trace record is processedin a serial manner. As soon as the indicator is encountered that all ofthe startup information has been processed, trace records from tracehooks and trace records from timer interrupts are then processed. Timerinterrupt information from the timer interrupt records are resolved withexisting hash tables. In addition, this information identifies thethread and function being executed. The data is stored in hash tableswith a count identifying the number of timer tick occurrences associatedwith each way of looking at the data. After all of the trace records areprocessed, the information is formatted for output in the form of areport.

Alternatively, trace information may be processed on-the-fly so thattrace data structures are maintained during the profiling phase. Inother words, while a profiling function, such as a timer interrupt, isexecuting, rather than (or in addition to) writing trace records to abuffer or file, the trace record information is processed to constructand maintain any appropriate data structures.

For example, during the processing of a timer interrupt during theprofiling phase, a determination could be made as to whether the codebeing interrupted is being interpreted by the Java interpreter. If thecode being interrupted is interpreted, the method ID of the method beinginterpreted may be placed in the trace record. In addition, the name ofthe method may be obtained and placed in the appropriate B-tree. Oncethe profiling phase has completed, the data structures may contain allthe information necessary for generating a profile report without theneed for post-processing of the trace file.

With reference now to FIG. 6, a flowchart depicts a process used by atrace program for generating trace records from processes executing on adata processing system. FIG. 6 provides further detail concerning thegeneration of trace records that were not described with respect to FIG.5.

Trace records may be produced by the execution of small pieces of codecalled “hooks”. Hooks may be inserted in various ways into the codeexecuted by processes, including statically (source code) anddynamically (through modification of a loaded executable). This processis employed after trace hooks have already been inserted into theprocess or processes of interest. The process begins by allocating abuffer (step 600), such as buffer 404 in FIG. 4. Next, in the depictedexample, trace hooks are turned on (step 602), and tracing of theprocesses on the system begins (step 604). Trace data is received fromthe processes of interest (step 606). This type of tracing may beperformed during phases 500 and/or 502. This trace data is stored astrace records in the buffer (step 608). A determination is made as towhether tracing has finished (step 610). Tracing finishes when the tracebuffer has been filled or the user stops tracing via a command andrequests that the buffer contents be sent to file. If tracing has notfinished the process returns to step 606 as described above.

Otherwise, when tracing is finished, the buffer contents are sent to afile for post-processing (step 612). A report is then generated inpost-processing (step 614) with the process terminating thereafter.

Although the depicted example uses post-processing to analyze the tracerecords, the processes of the present invention may be used to processtrace information in real-time depending on the implementation.

With reference now to FIG. 7, a flowchart depicts a process that may beused during an interrupt handler trace hook.

The process begins by obtaining a program counter (step 700). Typically,the program counter is available in one of the saved program stackareas. Thereafter, a determination is made as to whether the code beinginterrupted is interpreted code (step 702). This determination may bemade by determining whether the program counter is within an addressrange for the interpreter used to interpret bytecodes. If the code beinginterrupted is interpreted, a method block address is obtained for thecode being interpreted. A trace record is then written (step 706). Thetrace record is written by sending the trace information to a traceprogram, such as trace program 400, which generates trace records forpost-processing in the depicted example. This trace record is referredto as an interrupt record, or an interrupt hook.

This type of trace may be performed during phase 502. Alternatively, asimilar process, i.e. determining whether code that was interrupted isinterpreted code, may occur during post-processing of a trace file.

A set of processes may be employed to obtain sample-based profilinginformation. As applications execute, the applications may beperiodically interrupted in order to obtain information about thecurrent runtime environment. This information may be written to a bufferor file for post-processing, or the information may be processedon-the-fly into data structures representing an ongoing history of theruntime environment. FIGS. 8 and 9 describe sample-based profiling inmore detail.

A sample-based profiler may obtain information from the stack of aninterrupted thread. The thread is interrupted by a software timerinterrupt available in many operating systems. The user of the tracefacility selects either the program counter option or the stack unwindoption, which may be accomplished by enabling one major code or anothermajor code, as described further below. This timer interrupt is employedto sample information from a call stack. By walking back up the callstack, a complete call stack can be obtained for analysis. A “stackwalk” may also be described as a “stack unwind”, and the process of“walking the stack” may also be described as “unwinding the stack.” Eachof these terms illustrates a different metaphor for the process. Theprocess can be described as “walking” as the process must obtain andprocess the stack frames step-by-step or frame-by-frame. The process mayalso be described as “unwinding” as the process must obtain and processthe stack frames that point to one another, and these pointers and theirinformation must be “unwound” through many pointer dereferences.

The stack unwind follows the sequence of functions/method calls at thetime of the interrupt. A call stack is an ordered list of routines plusoffsets within routines (i.e. modules, functions, methods, etc.) thathave been entered during execution of a program. For example, if routineA calls routine B, and then routine B calls routine C, while theprocessor is executing instructions in routine C, the call stack is ABC.When control returns from routine C back to routine B, the call stack isAB. For more compact presentation and ease of interpretation within agenerated report, the names of the routines are presented without anyinformation about offsets. Offsets could be used for more detailedanalysis of the execution of a program, however, offsets are notconsidered further herein.

Thus, during timer interrupt processing or at post-processing, thegenerated sample-based profile information reflects a sampling of callstacks, not just leaves of the possible call stacks, as in some programcounter sampling techniques. A leaf is a node at the end of a branch,i.e. a node that has no descendants. A descendant is a child of a parentnode, and a leaf is a node that has no children.

With reference now FIG. 8, a diagram depicts the call stack containingstack frames. A “stack” is a region of reserved memory in which aprogram or programs store status data, such as procedure and functioncall addresses, passed parameters, and sometimes local variables. A“stack frame” is a portion of a thread's stack that represents localstorage (arguments, return addresses, return values, and localvariables) for a single function invocation. Every active thread ofexecution has a portion of system memory allocated for its stack space.A thread's stack consists of sequences of stack frames. The set offrames on a thread's stack represent the state of execution of thatthread at any time. Since stack frames are typically interlinked (e.g.,each stack frame points to the previous stack frame), it is oftenpossible to trace back up the sequence of stack frames and develop the“call stack”. A call stack represents all not-yet-completed functioncalls—in other words, it reflects the function invocation sequence atany point in time.

Call stack 800 includes information identifying the routine that iscurrently running, the routine that invoked it, and so on all the way upto the main program. Call stack 800 includes a number of stack frames802, 804, 806, and 808. In the depicted example, stack frame 802 is. atthe top of call stack 800, while stack frame 808 is located at thebottom of call stack 800. The top of the call stack is also referred toas the “root”. The timer interrupt (found in most operating systems) ismodified to obtain the program counter value (pcv) of the interruptedthread, together with the pointer to the currently active stack framefor that thread. In the Intel architecture, this is typicallyrepresented by the contents of registers: EIP (program counter) and EBP(pointer to stack frame). By accessing the currently active stack frame,it is possible to take advantage of the (typical) stack frame linkageconvention in order to chain all of the frames together. Part of thestandard linkage convention also dictates that the function returnaddress be placed just above the invoked-function's stack frame; thiscan be used to ascertain the address for the invoked function. Whilethis discussion employs an Intel-based architecture, this example is nota restriction. Most architectures employ linkage conventions that can besimilarly navigated by a modified profiling interrupt handler.

When a timer interrupt occurs, the first parameter acquired is theprogram counter value. The next value is the pointer to the top of thecurrent stack frame for the interrupted thread. In the depicted example,this value would point to EBP 808 a in stack frame 808. In turn, EBP 808points to EBP 806 a in stack frame 806, which in turn points to EBP 804a in stack frame 804. In turn, this EBP points to EBP 802 a in stackframe 802. Within stack frames 802-808 are EIPs 802 b-808 b, whichidentify the calling routine's return address. The routines may beidentified from these addresses. Thus, routines are defined bycollecting all of the return addresses by walking up or backwardsthrough the stack.

With reference now to the FIG. 9, an illustration of a call stack isdepicted. A call stack, such as call stack 900 is obtained by walkingthe call stack. A call stack is obtained each time a periodic event,such as, for example, a timer interrupt occurs. These call stacks may bestored as call stack unwind trace records (also called merely “stackunwinds”) within the trace file for post-processing or may be processedon-the-fly while the program continues to execute.

In the depicted example, call stack 900 contains a pid 902, which is theprocess identifier, and a tid 904, which is the thread identifier. Callstack 900 also contains addresses addr1 906, addr2 908 . . . addrN 910.In this example, addr1 906 represents the value of the program counterat the time of the interrupt. This address occurs somewhere within thescope of the interrupted function, addr2 908 represents an addresswithin the process that called the function that was interrupted. ForIntel-processor-based data processing systems, it represents the returnaddress for that call; decrementing that value by 4 results in theaddress of the actual call, also known as the call-site. Thiscorresponds with EIP 808 b in FIG. 8; addrN 910 is the top of the callstack (EIP 802 b). The call stack that would be returned if the timerinterrupt interrupted the thread,whose call stack state is depicted inFIG. 8 would consist of: a pid, which is the process id of theinterrupted thread; a tid, which is the thread id for the interruptedthread; a pcv, which is a program counter value (not shown on FIG. 8)for the interrupted thread; EIP 808 b; EIP 806 b; EIP 804 b; and EIP 802b. In terms of FIG. 9, pcv=addr1, EIP 808 b=addr2, EIP 806 b=addr3, EIP804 b=addr4, EIP 802 b=addr5.

With reference now to FIG. 10A, a diagram of a program executionsequence along with the state of the call stack at each functionentry/exit point is provided. The illustration shows entries and exitsoccurring at regular time intervals, but this is only a simplificationfor the illustration. If each function (A, B, C, and X in the figure)were instrumented with entry/exit event hooks, then complete accountingof the time spent within and below each function would be readilyobtained. Note in FIG. 10A that at time 0, the executing thread is inroutine C. The call stack at time 0 is C. At time 1, routine C callsroutine A, and the call stack becomes CA and so on. It should be notedthat the call stack in FIG. 10A is a reconstructed call stack that isgenerated by processing the event-based trace records in a trace file tofollow such events as method entries and method exits.

The accounting technique and data structure are described in more detailfurther below. Unfortunately, this type of instrumentation can beexpensive, can introduce bias, and in some cases, can be difficult toapply. Sample-based profiling, during which the sampling would belimited to sampling the program's call stack, helps to alleviate theperformance bias and other complications that entry/exit hooks canproduce.

Consider FIG. 10B, in which the same program is executed but is beingsampled on a regular basis. In the example, the interrupt occurs at afrequency equivalent to two timestamp values. Each sample includes asnapshot of the interrupted thread's call stack. Not all call stackcombinations are seen with this technique—for example, note that routineX does not show up at all in the set of call stack samples in FIG. 10B.This is an acceptable limitation of sampling. The idea is that with anappropriate sampling rate (e.g., 30-1000 times per second), the callstacks in which most of the time is spent will be identified. Althoughsome call stacks are omitted, it is a minor issue provided these callstacks are combinations for which little time is consumed.

In the event-based traces, there is a fundamental assumption that thetraces contain information about routine entries and matching routineexits. Often, entry-exit pairs are nested in the traces because routinescall other routines. Time spent (or memory consumed) between entry intoa routine and exit from the same routine is attributed to that routine,but a user of a profiling tool may want to distinguish between timespent directly in a routine and time spent in other routines that itcalls.

FIG. 10C shows an example of the manner in which time may be expended bytwo routines: a program's “main” calls routine A at time “t” equal tozero; routine A computes for 1 ms and then calls routine B; routine Bcomputes for 8 ms and then returns to routine A; routine A computes for1 ms and then returns to “main”. From the point of view of “main”,routine A took 10 ms to execute, but most of that time was spentexecuting instructions in routine B and was not spent executinginstructions within routine A. This is a useful piece of information fora person attempting to optimize the example program. In addition, ifroutine B is called from many places in the program, it might be usefulto know how much of the time spent in routine B was on behalf of (orwhen called by) routine A and how much of the time was on behalf ofother routines.

A fundamental concept in the output provided by the methods describedherein is the call stack. The call stack consists of the routine that iscurrently running, the routine that invoked it, and so on all the way upto main. A profiler may add a higher, thread level with the pid/tid (theprocess IDs and thread IDs). In any case, an attempt is made to followthe trace event records, such as method entries and exits, as shown inFIG. 10A, to reconstruct the structure of the call stack frames whilethe program was executing at various times during the trace.

The post-processing of a trace file may result in a report consisting ofthree kinds of time spent in a routine, such as routine A: (1) basetime—the time spent executing code in routine A itself; (2) cumulativetime (or “CUM time” for short)—the time spent executing in routine Aplus all the time spent executing every routine that routine A calls(and all the routines they call, etc.); and (3) wall-clock time orelapsed time. This type of timing information may be obtained fromevent-based trace records as these records have timestamp informationfor each record.

A routine's cumulative time is the sum of all the time spent executingthe routine plus the time spent executing any other routine while thatroutine is below it on the call stack. In the example above in FIG. 10C,routine A's base time is 2 ms, and its cumulative time is 10 ms. RoutineB's base time is 8 ms, and its cumulative time is also 8 ms because itdoes not call any other routines. It should be noted that cumulativetime may not be generated if a call stack tree is being generatedon-the-fly—cumulative time may only be computed after the fact duringthe post-processing phase of a profile utility.

For wall-clock or elapsed time, if while routine B was running, thesystem fielded an interrupt or suspended this thread to run anotherthread, or if routine B blocked waiting on a lock or I/O, then routine Band all the entries above routine B on the call stack accumulate elapsedtime but not base or cumulative time. Base and cumulative time areunaffected by interrupts, dispatching, or blocking. Base time onlyincreases while a routine is running, and cumulative time only increaseswhile the routine or a routine below it on the call stack is running.

In the example in FIG. 10C, routine A's elapsed time is the same as itscumulative time—10 ms. Changing the example slightly, suppose there wasa 1 ms interrupt in the middle of B, as shown in FIG. 10D. Routine A'sbase and cumulative time are unchanged at 2 ms and 10 ms, but itselapsed time is now 11 ms.

Although base time, cumulative time and elapsed time were defined interms of processor time spent in routines, profiling is useful forattributing consumption of almost any system resource to a set ofroutines, as described in more detail below with respect to FIG. 11B.Referring to FIG. 10C again, if routine A initiated two disk I/O's, andthen routine B initiated three more I/O's when called by routine A,routine A's “base I/O's” are two and routine A's “cumulative I/O's” arefive. “Elapsed I/O's” would be all I/O's, including those by otherthreads and processes, that occurred between entry to routine A and exitfrom routine A. More general definitions for the accounting conceptsduring profiling would be the following: base—the amount of the trackedsystem resource consumed directly by this routine; cumulative—the amountof the tracked system resource consumed by this routine and all routinesbelow it on the call stack; elapsed—the total amount of the trackedsystem resource consumed (by any routine) between entry to this routineand exit from the routine.

With reference now to FIG. 11A, a diagram depicts a tree structuregenerated from trace data. This figure illustrates a call stack tree1100 in which each node in tree structure 1100 represents a functionentry point.

Additionally, in each node in tree structure 1100, a number ofstatistics are recorded. In the depicted example, each node, nodes1102-1108, contains an address (addr), a base time (BASE), cumulativetime (CUM) and parent and children pointers. As noted above, this typeof timing information may be obtained from event-based trace records asthese records have timestamp information for each record. The addressrepresents a function entry point. The base time represents the amountof time consumed directly by the thread executing this function. Thecumulative time is the amount of time consumed by the thread executingthis function and all functions below it on the,call stack. In thedepicted example, pointers are included for each node. One pointer is aparent pointer, a pointer to the node's parent. Each node also containsa pointer to each child of the node.

Those of ordinary skill in the art will appreciate that tree structure1100 may be implemented in a variety of ways and that many differenttypes of statistics may be maintained at the nodes other than those inthe depicted example.

The call stack is developed from looking back at all return addresses.These return addresses will resolve within the bodies of thosefunctions. This information allows for accounting discrimination betweendistinct invocations of the same function. In other words, if function Xhas 2 distinct calls to function A, the time associated with those callscan be accounted for separately. However, most reports would not makethis distinction.

With reference now to FIG. 11B, a call stack tree which reflects callstacks observed during a specific example of system execution will nowbe described. At each node in the tree, several statistics are recorded.In the example shown in FIG. 11B, the statistics are time-basedstatistics. The particular statistics shown include the number ofdistinct times the call stack is produced, the sum of the time spent inthe call stack, the total time spent in the call stack plus the time inthose call stacks invoked from this call stack (referred to ascumulative time), and the number of instances of this routine above thisinstance (indicating depth of recursion).

For example, at node 1152 in FIG. 11B, the call stack is CAB, and thestatistics kept for this node are 2:3:4:1. Note that call stack CAB isfirst produced at time 2 in FIG. 10A, and is exited at time 3. Callstack CAB is produced again at time 4, and is exited at time 7. Thus,the first statistic indicates that this particular call stack, CAB, isproduced twice in the trace. The second statistic indicates that callstack CAB exists for three units of time (at time 2, time 4, and time6). The third statistic indicates the cumulative amount of time spent incall stack CAB and those call stacks invoked from call stack CAB (i.e.,those call stacks having CAB as a prefix, in this case CABB). Thecumulative time in the example shown in FIG. 11B is four units of time.Finally, the recursion depth of call stack CAB is one, as none of thethree routines present in the call stack have been recursively entered.

Those skilled in the art will appreciate that the tree structuredepicted in FIG. 11B may be implemented in a variety of ways, and avariety of different types of statistics may be maintained at each node.In the described embodiment, each node in the tree contains data andpointers. The data items include the name of the routine at that node,and the four statistics discussed above. Of course, many other types ofstatistical information may be stored at each node. In the describedembodiment, the pointers for each node include a pointer to the node'sparent, a pointer to the first child of the node (i.e. the left-mostchild), a pointer to the next sibling of the node, and a pointer to thenext instance of a given routine in the tree. For example, in FIG. 11B,node 1154 would contain a parent pointer to node 1156, a first childpointer to node 1158, a next sibling pointer equal to NULL (note thatnode 1154 does not have a next sibling), and a next instance pointer tonode 1162. Those skilled in the art will appreciate that other pointersmay be stored to make subsequent analysis more efficient. In addition,other structural elements, such as tables for the properties of aroutine that are invariant across instances, e.g., the routine's name,may also be stored.

The type of performance information and statistics maintained at eachnode are not constrained to time-based performance statistics. Thepresent invention may be used to present many types of trace informationin a compact manner which supports performance queries. For example,rather than keeping statistics regarding time, tracing may be used totrack the number of Java bytecodes executed in each method (i.e.,routine) called. The tree structure of the present invention would thencontain statistics regarding bytecodes executed rather than time. Inparticular, the quantities recorded in the second and third categorieswould reflect the number of bytecodes executed rather than the amount oftime spent in each method.

Tracing may also be used to track memory allocation and deallocation.Every time a routine creates an object, a trace record could begenerated. The tree structure of the present invention would then beused to efficiently store and retrieve information regarding memoryallocation. Each node would represent the number of method calls, theamount of memory allocated within a method, the amount of memoryallocated by methods called by the method, and the number of methodsabove this instance (i.e., the measure of recursion). Those skilled inthe art will appreciate that the tree structure of the present inventionmay be used to represent a variety of performance data in a manner whichis very compact, and allows a wide variety of performance queries to beperformed.

The tree structure shown in FIG. 11B depicts one way in which data maybe pictorially presented to a user. The same data may also be presentedto a user in tabular form as shown in FIG. 12.

With reference now to FIG. 12, a call stack tree presented as a tablewill now be described. Note that FIG. 12 contains a routine, pt_pidtid,which is the main process/thread which calls routine C. Table 12includes columns of data for Level 1230, RL 1232, Calls 1234, Base 1236,Cum 1238, and Indent 1240. Level 1230 is the tree level (counting fromthe root as level 0) of the node. RL 1232 is the recursion level. Calls1234 is the number of occurrences of this particular call stack, i.e.,the number of times this distinct call stack configuration occurs. Base1236 is the total observed time in the particular call stack, i.e., thetotal time that the stack had exactly these routines on the stack. Cum1238 is the total time in the particular call stack plus deeper levelsbelow it. Indent 1240 depicts the level of the tree in an indentedmanner. From this type of call stack configuration information, it ispossible to infer each unique call stack configuration, how many timesthe call stack configuration occurred, and how long it persisted on thestack. This type of information also provides the dynamic structure of aprogram, as it is possible to see which routine called which otherroutine. However, there is no notion of time-order in the call stacktree. It cannot be inferred that routines at a certain level were calledbefore or after other routines on the same level.

The pictorial view of the call stack tree, as illustrated in FIG. 11B,may be built dynamically on-the-fly or built statically using a tracefile as input. FIG. 13 depicts a flow chart of a method for building acall stack tree using a trace file as input. In FIG. 13, the call stacktree is built to illustrate module entry and exit points.

With reference now to FIG. 13, it is first determined if there are moretrace records in the trace file (step 1350). If so, several pieces ofdata are obtained from the trace record, including the time, whether theevent is an enter or an exit, and the module name (step 1352). Next, thelast time increment is attributed to the current node in the tree (step1354). A check is made to determine if the trace record is an enter oran exit record (step 1356). If it is an exit record, the tree istraversed to the parent (using the parent pointer), and the current treenode is set equal to the parent node (step 1358). If the trace record isan enter record, a check is made to determine if the module is already achild node of the current tree node (step 1360). If not, a new node iscreated for the module and it is attached to the tree below the currenttree node (step 1362). The tree is then traversed to the module's node,and the current tree node is set equal to the module node (step 1364).The number of calls to the current tree node is then incremented (step1366). This process is repeated for each trace record in the traceoutput file until there are no more trace records to parse (step 1368).

With reference now to FIG. 14, a flow chart depicts a method forbuilding a call stack tree dynamically as tracing is taking place duringsystem execution. In FIG. 14, as an event is logged, it is added to thetree in real time. Preferably, a separate call stack tree is maintainedfor each thread. The call stack tree reflects the call stacks recordedto date, and a current tree node field indicates the current location ina particular tree. When an event occurs (step 1470), the thread ID isobtained (step 1471). The time, type of event (i.e., in this case,whether the event is a method entry or exit), the name of the module(i.e., method), location of the thread's call stack, and location of thethread's “current tree node” are then obtained (step 1472). The lasttime increment is attributed to the current tree node (step 1474). Acheck is made to determine if the trace event is an enter or an exitevent (step 1476). If it is an exit event, the tree is traversed to theparent (using the parent pointer), and the current tree node is setequal to the parent node (step 1478). At this point, the tree can bedynamically pruned in order to reduce the amount of memory dedicated toits maintenance (step 1479). Pruning is discussed in more detail below.If the trace event is an enter event, a check is made to determine ifthe module is already a child node of the current tree node (step 1480).If not, a new node is created for the module, and it is attached to thetree below the current tree node (step 1482). The tree is then traversedto the module's node, and the current tree node is set equal to themodule node (step 1484). The number of calls to the current tree node isthen incremented (step 1486). Control is then passed back to theexecuting module, and the dynamic tracing/reduction program waits forthe next event to occur (step 1488).

One of the advantages of using the dynamic tracing/reduction techniquedescribed in FIG. 14 is its enablement of long-term system tracecollection with a finite memory buffer. Very detailed performanceprofiles may be obtained without the expense of an “infinite” tracebuffer. Coupled with dynamic pruning, the method depicted in FIG. 14 cansupport a fixed-buffer-size trace mechanism.

The use of dynamic tracing and reduction (and dynamic pruning in somecases) is especially useful in profiling the performance characteristicsof long running programs. In the case of long running programs, a finitetrace buffer can severely impact the amount of useful trace informationthat may be collected and analyzed. By using dynamic tracing andreduction (and perhaps dynamic pruning), an accurate and informativeperformance profile may be obtained for a long running program.

Many long-running applications reach a type of steady-state, where everypossible routine and call stack is present in the tree and updatingstatistics. Thus, trace data can be recorded and stored for suchapplications indefinitely within the constraints of a bounded memoryrequirement using dynamic pruning. Pruning has value in reducing thememory requirement for those situations in which the call stacks areactually unbounded. For example, unbounded call stacks are produced byapplications that load and run other applications.

Pruning can be performed in many ways, and a variety of pruning criteriais possible. For example, pruning decisions may be based on the amountof cumulative time attributed to a subtree. Note that pruning may bedisabled unless the amount of memory dedicated to maintaining the callstack exceeds some limit. As an exit event is encountered (such as step1478 in FIG. 14), the cumulative time associated with the current nodeis compared with the cumulative time associated with the parent node. Ifthe ratio of these two cumulative times does not exceed a pruningthreshold (e.g., 0.1), then the current node and all of its descendantsare removed from the tree. The algorithm to build the tree proceeds asbefore by traversing to the parent, and changing the current node to theparent.

Many variations of the above pruning mechanism are possible. Forexample, the pruning threshold can be raised or lowered to regulate thelevel of pruning from very aggressive to none. More global techniquesare also possible, including a periodic sweep of the entire call stacktree, removing all subtrees whose individual cumulative times are not asignificant fraction of their parent node's cumulative times.

Data reduction allows analysis programs to easily and quickly answermany questions regarding how computing time was spent within the tracedprogram. This information may be gathered by “walking the tree” andaccumulating the data stored at various nodes within the call stacktree, from which it can be determined the amount of time spent strictlywithin routine A, the total amount of time spent in routine A and in theroutines called by routine A either directly or indirectly, etc.

With reference now to FIG. 15, a diagram of a record generated using theprocesses of present invention is depicted. Each routine in record 1500is listed separately, along with information regarding the routine inFIG. 15. For example, calls column 1504 lists the number of times eachroutine has been called. BASE column 1506 contains the total time spentin the routine, while CUM column 1508 includes the cumulative time spentin the routine and all routines called by the routine. Name column 1512contains the name of the routine.

With reference now to FIG. 16, a diagram of another type of report thatmay be produced is depicted. The report depicted in FIG. 16 illustratesmuch of the same information found in FIG. 15, but in a slightlydifferent format. As with FIG. 15, diagram 1600 includes information oncalls, base time, and cumulative time.

FIG. 16 shows a trace output containing times spent within variousroutines as measured in microseconds. FIG. 16 contains one stanza(delimited by horizontal lines) for each routine that appears in thetrace output. The stanza contains information about the routine itselfon the line labeled “Self”, about who called it on lines labeled“Parent”, and about who the routine called on lines labeled “Child”. Thestanzas are in order of cumulative time. The third stanza is aboutroutine A, as indicated by the line beginning with “Self.” The numberson the “Self” line of this stanza show that routine A was called threetimes in this trace, once by routine C and twice by routine B. In theprofile terminology, routines C and B are (immediate) parents of routineA. Routine A is a child of routines C and B. All the numbers on the“Parent” rows of the second stanza are breakdowns of routine A'scorresponding numbers. Three microseconds of the seven microsecond totalbase time spent in A was when it was called by routine C, and threemicroseconds when it was first called by routine B, and another onemicrosecond when it was called by routine B for a second time. Likewise,in this example, half of routine A's fourteen microsecond cumulativetime was spent on behalf of each parent.

Looking now at the second stanza, we see that routine C called routine Band routine A once each. All the numbers on “Child” rows are subsets ofnumbers from the child's profile. For example, of the three calls toroutine A in this trace, one was by routine C; of routine A's sevenmicrosecond total base time, three microseconds were while it was calleddirectly by routine C; of routine A's fourteen microsecond cumulativetime, seven microseconds was on behalf of routine C. Notice that thesesame numbers are the first row of the third stanza, where routine C islisted as one of routine A's parents.

The four relationships that are true of each stanza are summarized atthe top of FIG. 16. First, the sum of the numbers in the Calls columnfor Parents equals the number of calls on the Self row. Second, the sumof the numbers in the Base column for Parents equals Self's base. Third,the sum of the numbers in the Cum column for Parents equals Self's Cum.These first three invariants are true because these characteristics arethe definition of Parent; collectively they are supposed to account forall of Self's activities. Fourth, the Cum in the Child rows accounts forall of Self's Cum except for its own Base.

Program sampling may contain information from the call stack and mayprovide a profile reflecting the sampling of an entire call stack, notjust the leaves. Furthermore, the sample-based profiling technique mayalso be applied to other types of stacks. For example, with Javaprograms, a large amount of time is spent in a routine called the“interpreter”. If only the call stack was examined, the profile wouldnot reveal much useful information. Since the interpreter also tracksinformation in its own stack, e.g., a Java stack (with its own linkageconventions), the process can be used to walk up the Java stack toobtain the calling sequence from the perspective of the interpreted Javaprogram.

With reference now to FIG. 17, a figure depicts a report generated froma trace file containing both event-based profiling information, such asmethod entry/exits, and stack unwind information generated duringsample-based profiling. FIG. 17 is similar to FIG. 12, in which a callstack tree is presented as a report, except that FIG. 17 containsembedded stack walking information. Call stack tree 1700 contains twostack unwinds generated within the time period represented by the totalof 342 ticks. Stack unwind identifier 1702 denotes the beginning ofstack unwind information 1706, with the names of routines that areindented to the right containing the stack information that the stackwalking process was able to discern. Stack unwind identifier 1704denotes the beginning of stack unwind information 1708. In this example,“J:” identifies an interpreted Java method and “F:” identifies a nativefunction, such as a native function within JavaOS. A call from a Javamethod to a native method is via “ExecuteJava.” Hence, at the point atwhich the stack walking process reaches a stack frame for an“ExecuteJava,” it cannot proceed any further up the stack as the stackframes are discontinued. The process for creating a tree containing bothevent-based nodes and sample-based nodes is described in more detailfurther below. In this case, identifiers 1702 and 1704 also denote themajor code associated with the stack unwind.

With reference now to FIG. 18A-18B, tables depict major codes and minorcodes that may be employed to instrument software modules for profiling.A set of codes may be used to turn on and off various types of profilingfunctions in a particular profiling session.

For example, as shown in FIGS. 18A-18B, the minor code for a stackunwind is designated as 0×7fffffff, which may be used for two differentpurposes. The first purpose, denoted with a major code of 0×40, is for astack unwind during a timer interrupt. The second purpose, denoted witha major code of 0×41, is for a stack unwind in an instrumented routine.When the stack information is output into a trace file with its majorand minor codes, the trace information that appears within the file canbe analyzed in the appropriate manner indicated by the major and minorcodes.

Other examples in the table show a profile or major code purpose oftracing jitted methods with a major code value of 0×50. Tracing ofjitted methods may be distinguished based on the minor code thatindicates method invocation or method exit. In contrast, a major code of0×30 indicates a profiling purpose of instrumenting interpreted methods,while the minor code again indicates, with the same values, methodinvocation or method exit.

Referring back to FIG. 17, the connection can be made between the use ofmajor and minor codes, the instrumentation of code, and thepost-processing of profile information. In the generated report shown inFIG. 17, the stack unwind identifiers can be seen to be equal to 0×40,which, according to the tables in FIGS. 18A-18B, is a stack unwindgenerated in response to a timer interrupt. This type of stack unwindmay have occurred in response to an interrupt that was created in orderto generate a sampled profile of the executing software.

As noted in the last column of the tables in FIGS. 18A-18B, by using autility that places a hook into a software module to be profiled, astack unwind may be instrumented into a routine. If so, the output forthis type of stack unwind will be designated with a major code of 0×41.

While profiling an executable program, the profiler would ideallyreceive pairs of corresponding entry and exit events. However, this isoften not the case. In a typical example, tracing may commence after aportion of the executable program has already been run. This may occurif the mechanism for initiating the profiler is activated in a dynamicmanner. In this case, exit events may be received by the profilerwithout having previously received corresponding entry trace records,i.e. without having previously received a matching entry event. The exitevent can be said to be “mismatched”.

As an additional example, there may be several execution paths thatoccur during a profiling session that are not correctly instrumented asa practical matter. This often occurs within exception paths. In thecase of exception paths, the profiler will receive entry events withoutcorresponding exit events because an exception path inherently causes anabrupt exit from a routine or method.

In yet another example of unmatched events, the instrumentation of theexecutable code may not be performed automatically. In order to useevent-based profiling, an executable program must be instrumented withentry and exit trace hooks. In the case of manually instrumented code,it is possible that a programmer can make a mistake and not placecorresponding entry and exit trace hooks into a routine. In this case,it is possible for the profiler to receive an exit event for a routinewithout a corresponding entry trace record.

In each of these examples, it is important to determine the executionpath flow and to construct a reasonable representation of the executionpaths. One may attempt to detect and recover from these types of errorswithin the profiler. Trace processing may continue for all error cases,and in general, reasonable execution paths are determined for bothcommon and unusual problems.

Error recovery processing may be accomplished both in real-time traceprocessing and in a post-processing phase of outputted trace data.During real-time trace processing, a call stack tree may be constructed,as described above with respect to FIG. 11A and FIG. 11B. At any timeduring the construction of the call stack tree, a pointer is maintainedto a current node that represents the routine that was last identifiedto be executing.

If the event that is currently being processed is an entry event, it maybe assumed that the entry event represents a call from the routinerepresented by the current node. Since it is not yet possible todetermine whether an error has occurred in which an exit event may bemissing in the execution flow preceding the event currently beingprocessed, entry events are assumed not to contain an error condition.Although an error condition may subsequently be detected, processing forentry events proceeds in the same manner whether or not an errorcondition may have occurred within the execution flow represented by thegenerated events.

While processing an exit event, it is assumed that the exit eventrepresents an exit from the current routine as represented in the callstack tree structure. However, the exit event may identify a routinethat does not match the current tree node, i.e. the exit event and thecurrent node in the call stack tree do not match, and the exit event issaid to be “mismatched”. Hence, at least one entry event is missing inthe trace data. Although it is not possible to determine how many eventsmay be missing from the trace data, one may attempt to determine a nodewithin the call stack tree that represents the exit event that iscurrently being processed.

As an ancillary comment, it should be noted that when an entry event orexit event is received, a current timestamp (or other execution-relatedcharacteristic) may be obtained, and the time increment (or otherexecution-related metric) between the event currently being processedand the previous event is applied to the current node. This default ruleworks reasonably well, even in those cases in which an error is detectedwith mismatched entry and exit events, because the current noderepresents the last known routine to be executing, and at least some ofthe time (or other execution-related metric) between the last event andthe event currently being processed has been spent within the currentnode.

With reference now to FIG. 19, a flowchart depicts the processing ofexit events, including error recovery processing, in an execution flowthat may contain errors in the form of missing or mismatched events. Theprocess shown in the flowchart assumes that an event which is currentlybeing processed has been determined to be an exit event. In this manner,the process shown in FIG. 19 may replace step 1478 shown in FIG. 14.

In this example, X is the module name of the module being exited by theexit event, Y is the module name of the current node of the call tree,and R is the module name of the root node. Certain relationships applyto the tree nodes, such as the parent of NODE is PAR(NODE), and thechild of NODE is CHILD(NODE).

The process in FIG. 19 begins with a determination of whether theexiting module X is the same as the current node Y, i.e. whether themodule name of exiting module X corresponds to the module nameassociated with current node Y (step 1902). If so, this is the normalexit with no error. The tree is popped, i.e., traversed to the currentnode's parent node (step 1904), and the process is complete with respectto the exit event which has just been processed.

If exiting module X is not equal to current node Y, then a determinationis made as to whether the current node Y is equal to the root node R(step 1906). This case is the most normal error case, which would occurwhen the entry event corresponding to the exit event currently beingprocessed occurred before the trace was turned on. If the current nodeis equal to the root node, then the tree is “repaired” by renaming theroot node R to exiting module X and allowing module x to inherit all ofthe metrics existing in root node R at this time (step 1908). A new nodeis then created for the root of the tree and attached as the parent ofthe node for exiting module X, i.e. PAR(X)=R and CHILD(R)=X (step 1910).The tree is now in the same structural shape as if the trace startedwith an entry to module X. The process is then complete with respect toprocessing the exit event when the current node is the root node.

If the current node is not equal to the root node, then the tree ispopped, i.e. the current node indicator or pointer points to the parentof the current node Y (step 1912). A determination is made as to whetherthe exiting module X is the same as the current node Y, i.e. whether themodule name of exiting module X corresponds to the module nameassociated with current node Y (step 1914). If so, then processingproceeds to step 1904 to complete the processing of the exit eventrecord that is currently being processed. In this situation, one or moretree pops were required to find a matching exit node. This can happendue to one of the following: (1) a long “jump” construct which is usedto return from a nested error condition; (2) an uninstrumented exceptionprocessing path; or (3) faulty instrumentation resulting in one or moremissing exit hooks.

If exiting module X is not equal to current node Y, then a determinationis made as to whether the current node Y is equal to the root node R(step 1916). If not, then the process loops back to step 1912. If so,then the tree is “repaired” by branching to steps 1908 and 1910.However, the interpretation of the situation to be repaired is differentthan previously described. The determination in step 1916 detects asevere error, possibly due to instrumentation errors or trace gatheringdifficulties, and a severe error message may be issued as appropriate.

In this manner, the process described in FIG. 19 may be used toconstruct a call stack tree representation that provides usefulinformation even though some of the nodes do not represent an exactcorrespondence to the execution flow of the executing program.

As noted above, tracing may commence after a portion of the executableprogram has already been run, which may cause missing entry and exitevents in the trace flow, and when the profiler receives an event, theevent does not match a node in the call stack tree. While this type ofevent mismatch may occur at the start of a profiling phase when tracingof a program commences, this type of event mismatch may also occurduring sample-based profiling in accordance with a preferred embodimentof the present invention—if the generation of trace data has been pausedand then resumed, then mismatched entry and exit events may appear inthe execution flow. However, the process of pausing and resumingevent-based profiling, i.e. sample-based profiling of events, hascertain benefits compared to other manners of profiling.

Creating a call stack tree during continuous event-based profilingcauses significant system performance degradation due to the treebuilding and metric accumulation activity associated with the profilingprocess. Sample-based profiling using stack unwinds is not alwaysreliable and may have inherent implementation problems, such as theinability to follow a pointer chain in the call stack or the profiledprogram code does not follow standard conventions. Even if the stackunwinds are reliable, they do not provide information about the lengthof execution for the routines being profiled. Hence, periodic samplingof entry/exit hooks provides some structural and quantitative measuresof a program's execution with less system performance degradation thanfully continuous event-based profiling and without total reliance onstack unwinds.

With reference now to FIG. 20, a flowchart depicts a process forsample-based profiling of entry/exit events employing user-specifiedsample metrics to build independent call stack tree segments. Theprocess assumes that the described actions are performed on a per threadbasis, i.e. the profiler would perform similar actions or processing foreach unique pidtid value, i.e. call stack tree segments are not onlyidentifiable by sample period but are also identifiable by pidtid value.In addition, the profiler may maintain and increment a sequence valuefor the sample period that provides an ordinal number for the sampleperiods.

The process begins with the profiler accepting user-specified metricvalues to be used by the profiler to control sample-based profiling(step 2002). These metric values may be stored in environment variables,input through command line parameters, etc. Alternatively, the profilercontains default metric values so that the user is not required to inputany metric values. At some point in time, the profiling phase of aprogram is eventually initiated (step 2004).

The profiler then disables tracing for a user-specified metric value(step 2006), and the profiler does not perform any trace processingduring this period (step 2008). After this period, the profiler wakes upand enables real-time trace processing (step 2010). The profiler buildsa new, independent call stack tree segment using the entry/exit eventswith error recovery processing during a period measured according toanother user-specified metric value (step 2012).

A determination is then made as to whether the program execution hascompleted (step 2014), and if not, then the process loops back to step2006 to continue profiling the execution of the program. If programexecution is complete, then the profiler outputs the call stack treesegments generated during the profiling phase (step 2016). During thepost-processing phase, the call stack tree segments are merged into asingle call stack tree representation (step 2018), and the process isthen complete. “Merging” a tree is defined as traversing a first tree tocreate a second tree which contains the union of the first tree and thesecond tree. In this case, all of the call stack tree segments aremerged into a single tree, i.e. a single tree is generated through theunion of all of the call stack tree segments.

Each call stack tree segment is a unique recording of the callingsequence between routines during its sample period. A call stack treesegment is properly classifiable as a tree data structure. Anindependent tree is generated for each periodic sample period, whichcreates a forest of trees based on pidtid values over time. Althougheach sample period creates an individual call stack tree, the term “callstack tree segment” is used to emphasize that each of these call stacktrees from these periods may be relatively small compared to a full callstack tree that records the calling sequences between routines for anentire profiling run of an executable program. The term “call stack treesegment” is also used to emphasize that these call stack trees aremerely a means to the end of eventually producing a resulting call stacktree for all of the sample periods. Hence, the information in a callstack tree segment is merely a “segment” of the profile information forone sample period compared to all of the profile information that isgenerated during the profiling of the program.

A call stack tree segment from a sample period is termed “independent”in that each call stack tree segment is considered to have its own rootand its own nodes representing the progression of routine calls ormethod invocations between the routines as represented in the treestructure. However, the profiler may maintain another data structurewhich associatively stores these call stack tree segments—“independence” in this context does not require that each data structureis not linked to another data structure.

Since the call stack tree segments are identifiable by pidtid values,the profiler may maintain some type of data structure for each pidtid.For example, the profiler may maintain a linked list in which eachelement in the linked list is the root node of a completed call stacktree segment. In other words, related call stack tree segments may beassociatively linked as required by the profiler to maintain and trackthe call stack tree segments.

The user may identify a sample time and number of events for eachsample. During each sample, a call stack tree is generated. After theprofiling phase is complete, the profiler may hold thousands of thesetrees, i.e. a. “forest” of trees. Alternatively, the profiler mayperiodically output the call stack trees into a file forpost-processing. This forest of trees may then be merged duringpost-processing.

For example, the user may specify a two second waiting or pausing periodand a one thousand event tracing period. Therefore, every two seconds,the profiler should take one thousand event snapshots. After a thousandoccurrences, the profiler would disable entry/exit event processing andwait for two seconds. After the profiler determines to begin traceprocessing again, which may occur after a timer interrupt, the profilerenables the entry/exit event processing and allows the event processingto continue. The profiler loops in this manner until the execution ofthe job is complete.

Alternatively, the sample could be defined in terms of time durationinstead of events. For example, the user could specify a ten millisecondsample duration every second. In a preferred embodiment, the user mayspecify several types of execution-related metrics for either thewaiting period and/or the tracing period. In addition, the profiler maymonitor several different execution-related characteristics other thanexecution time in each routine.

With reference now to FIG. 21, a flowchart depicts the process by whichone tree, called the source tree, is added to a second tree, called thetarget tree, to obtain a union between the two trees. The intent is tocopy the source tree into the target tree starting from the root of thetarget tree. If any part of the source tree starting at its root alreadyexists in the target tree as a child of its root, then the existingnodes in the target tree must be reused, increasing their profilingmetrics, instead of creating new, duplicate nodes in the target tree.

Each source tree node, S, has a unique parent node, par(S) in the sourcetree by the definition of “tree”, except for the root node, S_(root),which has no parent. Also each S will eventually have a uniquecorresponding node T(S) in the target tree. Each S is copied to thetarget tree as a child of its parent's unique corresponding node in thetarget tree. In other words, if S₁ is a child of S₂ then T(S₁) will be achild of T(S₂). Thus, par(S) must have been copied into the target treebefore S can be copied. All profile metrics for the source node areadded directly to the corresponding target node's fields. This includesthe number of calls, the CUM time, and the BASE time. However, to getthe copying process started, this requirement is waived for the root ofthe source tree; assume T(par(S_(root))) is the root of the target tree.The profile metrics from the root node of the source tree are added tothe root node of the target tree, and the root node of the source treeis marked as processed (step 2102).

Any node S from the source tree whose parent has already been copied tothe target tree can be selected as the next node to be copied (step2104). A determination is then made as to whether T(par(S)) already hasa child node representing the same routine as S (step 2106). If not, anew child node is created in the target tree (step 2108), the routinename of the new node is set to the routine name of S (step 2110), andthe profile metrics in the target node, such as the number of calls tothe target node, is set to zero (step 2112). The profile metrics in S,such as the number of calls to S, are then added to the profile metricsstored in its corresponding node T(S) (whether newly created orpre-existing) in the target tree, and the source node is marked asprocessed (step 2114).

A determination is then made as to whether there are any remaininguncopied nodes in the source tree (step 2116). If so, then the processbranches back to 2104 where another node is selected to be copied. Ifnot, the process is complete. In this manner, each call stack treesegment is added to the resulting, or final, call stack tree.

The advantages of the present invention should be apparent withreference to the detailed description provided above. With the profilingapproach described above, each new periodic sample is treated as anindependent tree segment. When the job is completed, each call stacktree segment in the forest of trees generated during the entireprofiling job are merged. During the merging operation, profile metrics,such as base times, are updated as the sums of base times for theroutines as appropriate. Almost no system perturbation or systemperformance degradation is caused outside of the sample periods whentrace processing is not occurring.

It is possible that call stack tree segments may be merged togetherincorrectly since the error recovery procedure cannot guarantee accuracyin the formation of call stack tree segments that are almost certain tohave errors in the representations of execution flows from unmatched ormismatched entry/exit events at the beginning of a sample period. Inother words, identical call stack trees that were generated in differentcontexts may be combined. However, the total time spent within a givenmethod is correct within the sampled context in which the time valueswere generated.

It is important to note that while the present invention has beendescribed in the context of a fully functioning data processing system,those of ordinary skill in the art will appreciate that the processes ofthe present invention are capable of being distributed in the form of acomputer readable medium of instructions and a variety of forms and thatthe present invention applies equally regardless of the particular typeof signal bearing media actually used to carry out the distribution.Examples of computer readable media include recordable-type media such afloppy disc, a hard disk drive, a RAM, and CD-ROMs and transmission-typemedia such as digital and analog communications links.

The description of the present invention has been presented for purposesof illustration and description, but is not intended to be exhaustive orlimited to the invention in the form disclosed. Many modifications andvariations will be apparent to those of ordinary skill in the art. Theembodiment was chosen and described in order to best explain theprinciples of the invention, the practical application, and to enableothers of ordinary skill in the art to understand the invention forvarious embodiments with various modifications as are suited to theparticular use contemplated.

What is claimed is:
 1. A method in a data processing system forprofiling a gram executing in the data processing system, the methodcomprising: performing sample-based profiling of the executing program;for each sample period, generating a tree data structure in which nodesof the tree data structure represent routines of the program thatexecute during the sample period; and in response to a determination ofcompletion of the execution of the program, merging tree data structuresfrom each sample period into a resulting tree data structure.
 2. Themethod of claim 1 further comprising: recording execution-relatedmetrics for a sample period in the nodes of the tree data structure. 3.The method of claim 1 further comprising: enabling event-based traceprocessing for a period measured by a first predetermined value.
 4. Themethod of claim 3 further comprising: in response to an occurrence of aselected event within an executing routine in the program during aperiod in which trace processing is enabled, recording trace datacomprising an indication of the executing routine; and representing theexecuting routine as a node in the tree data structure for the sampleperiod.
 5. The method of claim 3 wherein the first predetermined valueis selectable by a user.
 6. The method of claim 3 wherein the firstpredetermined value specifies a time period using a temporal metric. 7.The method of claim 3 wherein the first predetermined value specifies anumber of occurrences of events.
 8. The method of claim 3 furthercomprising: enabling trace processing in response to a selected timerinterrupt.
 9. The method of claim 1 further comprising: disablingevent-based trace processing for a period measured by a secondpredetermined value.
 10. The method of claim 9 wherein the secondpredetermined value is selectable by a user.
 11. The method of claim 9wherein the second predetermined value specifies a time period using atemporal metric.
 12. The method of claim 9 wherein the secondpredetermined value specifies a number of occurrences of events.
 13. Themethod of claim 1 further comprising: associatively maintaining the treedata structures from the sample periods by a process/thread identifier.14. The method of claim 1 further comprising: constructing the tree datastructures in real-time as trace data is generated during execution ofthe program.
 15. The method of claim 1 wherein the step of merging treedata structures further comprises: generating a root node for an emptytree data structure representing a final tree data structure; for eachtree data structure generated during a sample period, adding a tree datastructure for a sample period to a final tree data structure to create aunion of the tree data structure for a sample period and the final treedata structure.
 16. A data processing system for profiling a programexecuting in the data processing system, the data processing systemcomprising: performing means for performing sample-based profiling ofthe executing program; first generating means for generating, for eachsample period, a tree data structure in which nodes of the tree datastructure represent routines of the program that execute during thesample period; and merging means for merging, in response to adetermination of completion of the execution of the program, tree datastructures from each sample period into a resulting tree data structure.17. The data processing system of claim 16 further comprising: firstrecording means for recording execution-related metrics for a sampleperiod in the nodes of the tree data structure.
 18. The data processingsystem of claim 16 further comprising: enabling means for enablingevent-based trace processing for a period measured by a firstpredetermined value.
 19. The data processing system of claim 18 furthercomprising: second recording means for recording, in response to anoccurrence of a selected event within an executing routine in theprogram during a period in which trace processing is enabled, trace datacomprising an indication of the executing routine; and representingmeans for representing the executing routine as a node in the tree datastructure for the sample period.
 20. The data processing system of claim18 wherein the first predetermined value is selectable by a user. 21.The data processing system of claim 18 wherein the first predeterminedvalue specifies a time period using a temporal metric.
 22. The dataprocessing system of claim 18 wherein the first predetermined valuespecifies a number of occurrences of events.
 23. The data processingsystem of claim 18 further comprising: enabling means for enabling traceprocessing in response to a selected timer interrupt.
 24. The dataprocessing system of claim 16 further comprising: disabling means fordisabling event-based trace processing for a period measured by a secondpredetermined value.
 25. The data processing system of claim 24 whereinthe second predetermined value is selectable by a user.
 26. The dataprocessing system of claim 24 wherein the second predetermined valuespecifies a time period using a temporal metric.
 27. The data processingsystem of claim 24 wherein the second predetermined value specifies anumber of occurrences of events.
 28. The data processing system of claim24 further comprising: maintaining means for associatively maintainingthe tree data structures from the sample periods by a process/threadidentifier.
 29. The data processing system of claim 16 furthercomprising: constructing means for constructing the tree data structuresin real-time as trace data is generated during execution of the program.30. The data processing system of claim 16 wherein the merging means formerging tree data structures further comprises: second generating meansfor generating a root node for an empty tree data structure representinga final tree data structure; and adding means for adding, for each treedata structure generated during a sample period, a tree data structurefor a sample period to a final tree data structure to create a union ofthe tree data structure for a sample period and the final tree datastructure.
 31. A computer program product in a computer-readable mediumfor use in a data processing system for profiling an executing program,the computer program product comprising: first instructions forperforming sample-based profiling of the executing program; secondinstructions for generating, for each sample period, a tree datastructure in which nodes of the tree data structure represent routinesof the program that execute during the sample period; and thirdinstructions for merging, in response to a determination of completionof the execution of the program, tree data structures from each sampleperiod into a resulting tree data structure.
 32. The computer programproduct of claim 31 further comprising: instructions for enablingevent-based trace processing for a period measured by a firstpredetermined value.
 33. The computer program product of claim 32further comprising: instructions for recording, in response to anoccurrence of a selected event within an executing routine in theprogram during a period in which trace processing is enabled, trace dataincluding an indication of the executing routine; and instructions forrepresenting the executing routine as a node in the tree data structurefor the sample period.
 34. The computer program product of claim 32further comprising: instructions for enabling trace processing inresponse to a selected timer interrupt.
 35. The computer program productof claim 31 further comprising: instructions for disabling event-basedtrace processing for a period measured by a second predetermined value.36. The computer program product of claim 31 further comprising:instructions for associatively maintaining the tree data structures fromthe sample periods by a process/thread identifier.
 37. The computerprogram product of claim 31 wherein the instructions for merging treedata structures further comprises: instructions for generating a rootnode for an empty tree data structure representing a final tree datastructure; and instructions for adding, for each tree data structuregenerated during a sample period, a tree data structure for a sampleperiod to a final tree data structure to create a union of the tree datastructure for a sample period and the final tree data structure.